feat: add connector props and format options in rw_catalog #19689

tabVersion · 2024-12-05T08:02:31Z

I hereby agree to the terms of the RisingWave Labs, Inc. Contributor License Agreement.

What's changed and what's your intention?

resolve #19072

Checklist

I have written necessary rustdoc comments
I have added necessary unit tests and integration tests
I have added test labels as necessary. See details.
I have added fuzzing tests or opened an issue to track them. (Optional, recommended for new SQL features Sqlsmith: Sql feature generation #7934).
My PR contains breaking changes. (If it deprecates some features, please create a tracking issue to remove them in the future).
All checks passed in ./risedev check (or alias, ./risedev c)
My PR changes performance-critical code. (Please run macro/micro-benchmarks and show the results.)

My PR contains critical fixes that are necessary to be merged into the latest release. (Please check out the details)

Documentation

My PR needs documentation updates. (Please use the Release note section below to summarize the impact on users)

Release note

If this PR includes changes that directly affect users or other significant modifications relevant to the community, kindly draft a release note to provide a concise summary of these changes. Please prioritize highlighting the impact these changes will have on users.

tabVersion · 2024-12-05T08:03:31Z

for rw_catalog.rw_sources

id | name | schema_id | owner | connector |      columns      |      format       |    row_encode    | append_only | associated_table_id | connection_id |                                                                                             definition                                                                                             |              acl              |      initialized_at       |        created_at         |           initialized_at_cluster_version            |             created_at_cluster_version              | is_shared |                                                                                       connector_props                                                                                       |                          format_encode_options
----+------+-----------+-------+-----------+-------------------+-------------------+------------------+-------------+---------------------+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------+---------------------------+---------------------------+-----------------------------------------------------+-----------------------------------------------------+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------
  7 | t    |         2 |     1 | KAFKA     | {bar,foo,_row_id} | FORMAT_TYPE_PLAIN | ENCODE_TYPE_AVRO | t           |                   6 |               | CREATE TABLE t () WITH (connector = 'kafka', properties.bootstrap.server = '127.0.0.1:9092', topic = 'avro_alter_table_test') FORMAT PLAIN ENCODE AVRO (schema.registry = 'http://127.0.0.1:8081') | {postgres=r/root,root=r/root} | 2024-12-05 07:54:35+00:00 | 2024-12-05 07:54:35+00:00 | PostgreSQL 13.14.0-RisingWave-2.2.0-alpha (unknown) | PostgreSQL 13.14.0-RisingWave-2.2.0-alpha (unknown) | f         | {"connector":{"type":"plaintext","value":"kafka"},"properties.bootstrap.server":{"type":"plaintext","value":"127.0.0.1:9092"},"topic":{"type":"plaintext","value":"avro_alter_table_test"}}  | {"schema.registry":{"type":"plaintext","value":"http://127.0.0.1:8081"}}
 12 | t1   |         2 |     1 | KAFKA     | {bar,foo,_row_id} | FORMAT_TYPE_PLAIN | ENCODE_TYPE_AVRO | t           |                  11 |               | CREATE TABLE t1 () WITH (connector = 'kafka', properties.bootstrap.server = secret secret_1, topic = 'avro_alter_table_test') FORMAT PLAIN ENCODE AVRO (schema.registry = 'http://127.0.0.1:8081') | {postgres=r/root,root=r/root} | 2024-12-05 08:00:40+00:00 | 2024-12-05 08:00:40+00:00 | PostgreSQL 13.14.0-RisingWave-2.2.0-alpha (unknown) | PostgreSQL 13.14.0-RisingWave-2.2.0-alpha (unknown) | f         | {"connector":{"type":"plaintext","value":"kafka"},"properties.bootstrap.server":{"type":"secret","value":{"value":"secret_1"}},"topic":{"type":"plaintext","value":"avro_alter_table_test"}} | {"schema.registry":{"type":"plaintext","value":"http://127.0.0.1:8081"}}
 15 | t2   |         2 |     1 | KAFKA     | {bar,foo,_row_id} | FORMAT_TYPE_PLAIN | ENCODE_TYPE_AVRO | t           |                  14 |            13 | CREATE TABLE t2 () WITH (connector = 'kafka', connection = connection conn, topic = 'avro_alter_table_test') FORMAT PLAIN ENCODE AVRO (schema.registry = 'http://127.0.0.1:8081')                  | {root=r/root,postgres=r/root} | 2024-12-05 08:24:52+00:00 | 2024-12-05 08:24:52+00:00 | PostgreSQL 13.14.0-RisingWave-2.2.0-alpha (unknown) | PostgreSQL 13.14.0-RisingWave-2.2.0-alpha (unknown) | f         | {"connector":{"type":"plaintext","value":"kafka"},"properties.bootstrap.server":{"type":"secret","value":{"value":"secret_1"}},"topic":{"type":"plaintext","value":"avro_alter_table_test"}} | {"schema.registry":{"type":"plaintext","value":"http://127.0.0.1:8081"}}
(3 rows)

for rw_catalog.rw_sinks

id |  name  | schema_id | owner | connector |          sink_type          | connection_id |                                                                                   definition                                                                                    |              acl              |      initialized_at       |        created_at         |           initialized_at_cluster_version            |             created_at_cluster_version              |                                                                                 connector_props                                                                                  |                                                      format_encode_options
----+--------+-----------+-------+-----------+-----------------------------+---------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------------+---------------------------+---------------------------+-----------------------------------------------------+-----------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------
 10 | sink_1 |         2 |     1 | KAFKA     | SINK_TYPE_FORCE_APPEND_ONLY |               | CREATE SINK sink_1 FROM t1 WITH (connector = 'kafka', properties.bootstrap.server = secret secret_1, topic = 'demo_sink') FORMAT PLAIN ENCODE JSON (force_append_only = 'true') | {root=r/root,postgres=r/root} | 2024-12-05 08:21:28+00:00 | 2024-12-05 08:21:29+00:00 | PostgreSQL 13.14.0-RisingWave-2.2.0-alpha (unknown) | PostgreSQL 13.14.0-RisingWave-2.2.0-alpha (unknown) | {"connector":{"type":"plaintext","value":"kafka"},"properties.bootstrap.server":{"type":"secret","value":{"value":"secret_1"}},"topic":{"type":"plaintext","value":"demo_sink"}} | {"force_append_only":{"type":"plaintext","value":"true"},"timestamptz.handling.mode":{"type":"plaintext","value":"utc_string"}}

yezizp2012

Rest LGTM

src/frontend/src/catalog/system_catalog/rw_catalog/rw_sinks.rs

yezizp2012

Rest LGTM

src/frontend/src/catalog/system_catalog/rw_catalog/rw_sources.rs

fuyufjh · 2024-12-18T03:35:23Z

Did it include authorization? i.e. Can I (non-root) see a source/table/sink created by other users?

yezizp2012 · 2024-12-18T03:38:38Z

Did it include authorization? i.e. Can I (non-root) see a source/table/sink created by other users?

No for most system catalogs for now. I will add uniformly. #19722

source

2561ec6

github-actions bot added the type/feature label Dec 5, 2024

tabversion added 2 commits December 5, 2024 16:05

rename

7c48e61

sink

46320e4

tabVersion marked this pull request as ready for review December 5, 2024 08:26

tabVersion requested review from neverchanje, yezizp2012 and xuhui-lu December 5, 2024 08:26

rename

7e71244

yezizp2012 reviewed Dec 5, 2024

View reviewed changes

src/frontend/src/catalog/system_catalog/rw_catalog/rw_sinks.rs Outdated Show resolved Hide resolved

src/frontend/src/catalog/system_catalog/rw_catalog/rw_sinks.rs Outdated Show resolved Hide resolved

tabversion and others added 5 commits December 5, 2024 17:06

fix

a6f27ce

Merge branch 'main' into tab/rw-conn-props

3c9f7ce

Merge branch 'main' into tab/rw-conn-props

86a98d8

Merge branch 'main' into tab/rw-conn-props

b2f76b3

fix planner test

54d9d9a

yezizp2012 approved these changes Dec 10, 2024

View reviewed changes

src/frontend/src/catalog/system_catalog/rw_catalog/rw_sources.rs Outdated Show resolved Hide resolved

rename

eed8364

tabVersion enabled auto-merge December 10, 2024 11:16

Merge branch 'main' into tab/rw-conn-props

643979e

tabVersion added this pull request to the merge queue Dec 10, 2024

Merged via the queue into main with commit 4545b6e Dec 10, 2024
28 of 29 checks passed

tabVersion deleted the tab/rw-conn-props branch December 10, 2024 13:22

tabVersion mentioned this pull request Dec 18, 2024

fix: reduct credentials in rw_catalog #19845

Closed

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add connector props and format options in rw_catalog #19689

feat: add connector props and format options in rw_catalog #19689

tabVersion commented Dec 5, 2024

tabVersion commented Dec 5, 2024 •

edited

Loading

yezizp2012 left a comment

yezizp2012 left a comment

fuyufjh commented Dec 18, 2024

yezizp2012 commented Dec 18, 2024

feat: add connector props and format options in rw_catalog #19689

feat: add connector props and format options in rw_catalog #19689

Conversation

tabVersion commented Dec 5, 2024

What's changed and what's your intention?

Checklist

Documentation

Release note

tabVersion commented Dec 5, 2024 • edited Loading

yezizp2012 left a comment

Choose a reason for hiding this comment

yezizp2012 left a comment

Choose a reason for hiding this comment

fuyufjh commented Dec 18, 2024

yezizp2012 commented Dec 18, 2024

tabVersion commented Dec 5, 2024 •

edited

Loading